One-sided reliable remote direct memory operations

ABSTRACT

Techniques are provided to allow more sophisticated operations to be performed remotely by machines that are not fully functional. Operations that can be performed reliably by a machine that has experienced a hardware and/or software error are referred to herein as Remote Direct Memory Operations or “RDMOs”. Unlike RDMAs, which typically involve trivially simple operations such as the retrieval of a single value from the memory of a remote machine, RDMOs may be arbitrarily complex. The techniques described herein can help applications run without interruption when there are software faults or glitches on a remote system with which they interact.

BENEFIT CLAIM

This application claims the benefit as a Divisional of application Ser.No. 16/055,978, filed Aug. 6, 2018, the entire contents of which ishereby incorporated by reference as if fully set forth herein, under 35U.S.C. § 120. The applicant(s) hereby rescind any disclaimer of claimscope in the parent application(s) or the prosecution history thereofand advise the USPTO that the claims in this application may be broaderthan any claim in the parent application(s).

FIELD OF THE INVENTION

The present invention relates to computer systems and, morespecifically, to techniques for increasing availability ofremotely-accessed functionality.

BACKGROUND

One way to improve the availability of a service is to design theservice in such a way that it continues to function properly even whenone or more of its components fails. For example, U.S. patentapplication Ser. No. 15/606,322, filed May 26, 2017 (which isincorporated herein by this reference) describes techniques for enablinga requesting entity to retrieve data that is managed by a databaseserver instance from the volatile memory of a remote server machine thatis executing the database server instance without involving the databaseserver instance in the retrieval operation.

Because the retrieval does not involve the database server instance, theretrieval operation may succeed even when the database server instance(or the host server machine itself) has stalled or become unresponsive.In addition to increasing availability, direct retrieval of data willoften be faster and more efficient than retrieval of the sameinformation through conventional interaction with the database serverinstance.

To retrieve “target data” specified in a database command from a remotemachine without involving the remote database server instance, therequesting entity first uses Remote Direct Memory Access (RDMA) toaccess information about where the target data resides in the servermachine. Based on such target location information, the requestingentity uses RDMA to retrieve the target data from the host servermachine without involving the database server instance. The RDMA reads(data retrieval operations) issued by the requesting entity areunilateral operations and do not require CPU interruption or OS kernelinvolvement on the host server machine (RDBMS server).

RDMA techniques work well for operations that simply involve theretrieval of data from volatile memory of a crashed server machine.However, it is desirable to provide high availability even when thefailed component is responsible for performing an operation that is moresophisticated than a mere memory access. To address this need, somesystems provide a limited set of “verbs” for performing remoteoperations via the Network Interface Controller, such as memory accessesand atomic operations (test and set, compare and swap). These operationscan complete as long as the system is powered up and the NIC has accessto the host memory. However, the type of operations that they support islimited.

More sophisticated operations on data that reside in the memory of aremote machine are typically performed by making remote procedure calls(RPCs) to applications running on the remote machine. For example, adatabase client that desires the average of a set of numbers that isstored in a database may make an RPC to the remote database serverinstance that manages the database. In response to the RPC, the remotedatabase server instance reads the set of numbers, calculates theaverage, and sends the average back to the database client.

In this example, if the remote database server fails, theaverage-computing operation fails. However, it may be possible to useRDMA to retrieve from the remote server each number in the set. Onceeach number in the set of numbers is retrieved by the requesting entity,the requesting entity may perform the average-computing operation.However, using RDMA to retrieve each number in the set, and thenperforming the calculation locally, is far less efficient than havingthe application on the remote server retrieve the data and perform theaverage-computing operation. Thus, it is desirable to expand the scopeof operations that continue to be available from a remote server whenthe remote server is not fully functional.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a block diagram of a requesting entity interacting with aremote machine that includes several execution candidates for arequested operation, each of which is implemented in a distinctreliability domain, according to an embodiment;

FIG. 2 is a flowchart illustrating use of fallback execution candidates,according to an embodiment;

FIG. 3 is a flowchart illustrating use of parallel execution candidates,according to an embodiment; and

FIG. 4 is a block diagram of a computer system that may be used toimplement the techniques described herein.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

Techniques are described herein to allow more sophisticated operationsto be performed remotely by machines that are not fully functional.Operations that can be performed reliably by a machine that hasexperienced a hardware and/or software error are referred to herein asRemote Direct Memory Operations or “RDMOs”. Unlike RDMAs, whichtypically involve trivially simple operations such as the retrieval of asingle value from the memory of a remote machine, RDMOs may bearbitrarily complex. For example, an RDMO may cause a remote machine tocompute the average of a set of numbers, where the numbers reside in thememory of the remote machine. The techniques described herein can helpapplications run without interruption when there are software faults orglitches on a remote system with which they interact.

Execution Candidates and Reliability Domains

According to one embodiment, within a single machine, multiple entitiesare provided for executing the same RDMO. The entities, within a givenmachine, that are capable of executing a particular RDMO are referred toherein as the “execution candidates” for the RDMO.

According to one embodiment, although in the same machine, the executioncandidates for an RDMO belong to separate reliability domains. The“reliability domain” of an execution candidate generally refers to thesoftware/hardware that must be functioning correctly on the machine forthe execution candidate to perform the RDMO correctly. Two executioncandidates belong to different reliability domains if one of theexecution candidates is able to perform the RDMO correctly while asoftware and/or hardware error has rendered the other executioncandidate unable to perform the RDMO correctly.

Because multiple execution candidates of an RDMO belong to differentreliability domains, it is possible for one of the execution candidatesto perform the RDMO during periods in which hardware/software errorswithin the machine prevent others of the execution candidates in themachine from performing the RDMO. The fact that multiple executioncandidates are available for a particular RDMO increases the likelihoodthat the RDMO will succeed when requested by requesting entities that donot reside on the machine.

Increasing Availability of RDMOs

When a remote server is executing multiple execution candidates for aparticular RDMO, and the execution candidates are from distinctreliability domains, the availability of the RDMO is increased. Forexample, in one embodiment, when a requesting entity requests theparticular RDMO, an attempt is made to perform the particular RDMO usinga first execution candidate on the machine. If the first executioncandidate is unable to perform the RDMO, then an attempt is made toperform the particular RDMO using a second execution candidate on themachine. This process may continue until the particular RDMO succeeds,or all execution candidates have been tried.

In an alternative embodiment, when a requesting entity requests theparticular RDMO, the particular RDMO may be attempted concurrently bytwo or more of the execution candidates. If any of the executioncandidates succeeds, then it is reported to the requesting entity thatthe particular RDMO was successful.

In one embodiment, an execution candidate of an RDMO may be an entitythat has been dynamically programmed to perform the RDMO. For example, acompute unit in a network interface controller of a machine may executean interpreter. In response to determining that a particular RDMO shouldbe performed in the network controller, rather than by an applicationrunning on the machine, data comprising instructions for performing theparticular RDMO may be sent to the interpreter. In response tointerpreting those instructions at the network controller, theparticular RDMO may be performed even though the application itself mayhave crashed.

Reliability Domains

As mentioned above, the “reliability domain” of an execution candidategenerally refers to the software/hardware that must be functioningcorrectly for the execution candidate to successfully perform an RDMO.FIG. 1 is a block diagram of a system that includes a machine 100 inwhich execution candidates of an RDMO reside at multiple distinctreliability domains. Specifically, a requesting entity 110 may request aparticular RDMO by sending a request to machine 100 over network 112.Machine 100 includes a network interface controller 102 that has one ormore compute units (represented as processing unit 104) that isexecuting firmware and/or software loaded into its local volatile memory106.

Machine 100 includes a processor 120 that executes an operating system132 and any number of applications, such as application 134. The codefor the operating system 132 and application 134 may be stored on apersistent storage 136 and loaded into volatile memory 130 as needed.Processor 120 may be one of any number of processors within machine 100.Processor 120 may itself have many distinct compute units, shown ascores 122, 124, 126 and 128.

Processor 120 includes circuitry (illustrated as uncore 142) that allowsentities executing in network interface controller 102 to access data140 in the volatile memory 130 of machine 100 without involving cores122, 124, 126 and 128.

In a remote server such as machine 100, entities capable of performingan RDMO requested by requesting entity 110 may reside in any number ofreliability domains, including but not limited to any of the following:

-   -   hardware within network interface controller 102    -   an FPGA within network interface controller 102    -   firmware stored in network interface controller 102    -   software executed within network interface controller 102 by        processing unit 104    -   instructions in volatile memory 106 that are interpreted by an        interpreter that is being executed within network interface        controller 102 by processing unit 104    -   an operating system 132, loaded into volatile memory 130 of        machine 100, executed by one or more cores (122, 124, 126 and        128) of processor 120    -   an application 134, loaded into volatile memory 130 of machine        100, executed by one or more cores (122, 124, 126 and 128) of        processor 120    -   software executing in a privileged domain (such as “dom0”).        Privileged domains and dom0 are described, for example, at        en.wikipedia.org/wiki/Xen.

This list of reliability domains is merely exemplary, and the techniquesdescribed herein are not limited to execution candidates from thesereliability domains. The examples given above qualify as distinctreliability domains because entities within those domains fail underdifferent conditions. For example, operating system 132 may continue tofunction normally even when application 134 has crashed or otherwisefailed. Similarly, an execution candidate running within networkinterface controller 102 may continue to function normally even when allprocesses being executed by processor 120 (including operating system132 and application 134) have crashed.

Further, since each core within processor 120 is a compute unit that mayfail independently of the other compute units within processor 120, anexecution candidate being executed by core 122 is in a differentreliability domain than an execution candidate that is being executed bycore 124.

Adding Support for New RDMOs

Machine 100 may include special purpose hardware, either in the networkinterface controller 102 or elsewhere, to implement a particular RDMO.However, hardware-implemented execution candidates cannot easily beextended to support additional RDMOs. Therefore, according to oneembodiment, mechanisms are provided for adding support for additionalRDMOs to other types of execution candidates. For example, assume thatNIC 102 includes firmware for executing a particular set of RDMOs. Underthese conditions, support for additional RDMOs may be added byconventional firmware update techniques to NIC 102.

On the other hand, if the execution candidate is implemented in an FPGA,support for new RDMOs can be added by reprogramming the FPGA. Suchreprogramming may be performed, for example, by loading the FPGA with arevised FPGA bitstream at powerup.

Similarly, support for new RDMOs may be added to software-implementedexecution candidates (e.g. software in NIC 102, operating system 132,and application 134) using conventional software update techniques. Inan embodiment that involves executing an interpreter within NIC 102, newRDMOs can be supported by sending code which implements the new RDMOs toNIC 102. NIC 102 may store the code in volatile memory 106, and performthe new RDMOs by interpreting the code. For example, NIC 102 may beexecuting a Java Virtual Machine, and requesting entity 110 may causethe NIC 102 to perform a new RDMO (e.g. computing the average of a setof numbers) by sending Java byte-code to NIC 102 which, when interpretedby the Java Virtual Machine, causes the target set of numbers to beretrieved from volatile memory 130, and the average thereof to becomputed.

Example RDMOs

In the foregoing discussion, examples are given where the RDMO inquestion is computing the average of a set of values. This is merely oneexample of an RDMO that can be supported by execution candidates frommultiple distinct reliability domains. As mentioned above, RDMOs may bearbitrarily complex. However, the more complex the RDMO, the greater theresources that will be required to execute the RDMO efficiently, and thegreater the likelihood that an execution candidate for the RDMO willencounter problems. In addition, complex RDMOs may execute slowly whenperformed by execution candidates from reliability domains with limitedresources. For example, a complex RDMO that is normally performed byapplication 134 executed using all of the compute units of processor 120will take significantly longer if executed by a relatively“light-weight” processing unit 104 on NIC 102.

Examples of RDMOs that may be supported by multiple executioncandidates, from different reliability domains, within the same machine,include but are not limited to:

-   -   setting a batch of flags, within the remote server machine, to        indicate that corresponding portions of the remote server        machine's volatile memory are unavailable or corrupt    -   adjusting the state, within the remote server machine, to cause        the remote server machine to respond to certain types of        requests with error notifications. This may avoid the need to        inform all other computing devices when the remote server        machine is failing to handle certain requests properly.    -   performing a batch of changes to a secondary replica of a data        structure (stored in volatile and/or persistent memory at the        remote server machine) to cause the secondary replica to reflect        the changes made to a primary replica of the data structure        (stored in volatile and/or persistent memory of another        machine).    -   any algorithm involving branching (e.g. logic configured to test        one or more conditions and determine which way to branch (and        therefore which actions to perform) based on the outcome of the        test).    -   any algorithm involving multiple reads of and/or multiple writes        to, the volatile memory of the remote server. Because each read        and/or write does not involve an RDMA communication between        machines, executing multiple reads and/or writes in response to        a single RDMO request significantly reduces inter-machine        communication overhead.    -   reading data from/writing data to a persistent storage device        (e.g. storage 136) that is operatively coupled to the remote        server machine. The ability to do this using an execution        candidate in the NIC 102 is particularly useful when the        persistent storage is not directly accessible to the requesting        entity, so that data on persistent storage 136 remains available        to the requesting entity even when software running on the        compute units of the remote machine (e.g. cores 122, 124, 126        and 128), potentially including the operating system 132 itself,        may have crashed.

Fallback Execution Candidates

As mentioned above, availability of an RDMO may be increased by havingan RDMO execution candidate from one reliability domain serve as afallback for an RDMO execution candidate from another reliabilitydomain. The choice of which execution candidate is the primary candidatefor the RDMO and which is the fallback candidate, may vary based on thecharacteristics of the candidates and the nature of the RDMO.

For example, in the case where the RDMO is for a relatively simpleoperation, it is possible that the operation may be performed moreefficiently by an execution candidate running in NIC 102 than anapplication 134 running on processor 120. Thus, for that particularRDMO, it may be desirable to have the execution candidate running in NIC102 to be the primary execution candidate for the RDMO, and application134 be the fallback execution candidate. On the other hand, if the RDMOis relatively complex, it may be more efficient for application 134 tobe used as the primary execution candidate for the RDMO, while anexecution candidate on NIC 102 serves as the fallback executioncandidate.

FIG. 2 is a flowchart illustrating use of fallback execution candidates,according to an embodiment. At step 200, a requesting entity sends arequest for an RDMO over a network to a remote computing device. At step202, a first execution candidate at the remote computing device attemptsto perform the RDMO. If the RDMO is successfully performed (determinedat step 204), a response indicating the success (possibly containingadditional result data) is sent back to the requesting entity at step206. Otherwise, it is determined at step 208 whether another executioncandidate, which has not yet been tried, is available for executing theRDMO. If no other execution candidate is available for the RDMO, then anerror indication is sent back to the requesting entity at step 210.Otherwise, an attempt is made to perform the RDMO using anotherexecution candidate.

This process may be continued until the RDMO is successfully performed,or all execution candidates for the RDMO have failed. In one embodiment,an entity at the remote server is responsible for iterating through theexecution candidates. In such an embodiment, attempts to perform theRDMO using the fallback execution candidates are made without informingthe requesting entity of any failures until all possible executioncandidates for the RDMO have failed. This embodiment reduces that amountof inter-machine message traffic generated during the iteration process.

In an alternative embodiment, the requesting entity is informed eachtime an execution candidate fails to perform the RDMO. In response to afailure indication, the requesting entity determines which executioncandidate to try next, if any. In response to determining that aparticular execution candidate should be tried next, the requestingentity sends another request to the remote computing device. The newrequest indicates the fallback execution candidate that should thenattempt to perform the RDMO.

It should be noted that failure of an execution candidate may beimplicitly indicated, rather than explicitly indicated. For example, itmay be assumed that an execution candidate has failed if, for example,the execution candidate has not acknowledged success after a particularamount of time has elapsed. In such cases, the requesting entity mayissue a request that the RDMO be performed by another executioncandidate before receiving any explicit indication that apreviously-selected execution candidate has failed.

The actual execution candidates that attempt to perform any given RDMO,as well as the sequence in which the execution candidates make theattempts, may vary based on a variety of factors. In one embodiment, theexecution candidates are tried in an order that is based on theirlikelihood of failure, where an application running on the remotecomputing device (which may be the candidate most likely to crash) istried first (because it has more resources available), and ahardware-implemented candidate in the NIC (which may be the candidateleast likely to crash) is tried last (because it has the least resourcesavailable).

As another example, in situations where it appears that the processorsof the remote computing device are overloaded or crashed, the requestingentity may first request that the RDMO be performed by an entityimplemented in the network interface controller of the remote computingdevice. On the other hand, if there is no such indication, therequesting entity may first request that the RDMO be performed by anapplication, running on the remote machine, that can take full advantageof the computational hardware available thereon.

As another example, for relatively simple RDMOs, the requesting devicemay first request that the RDMOs be performed by relatively“light-weight” execution candidates implemented in the network interfacecontroller. On the other hand, requests for relatively complex RDMOs mayfirst be sent to applications on the remote computing device, and onlysent to light-weight execution candidates when the applications fail toperform the RDMOs.

Parallel Execution Candidates

As also mentioned above, availability of an RDMO may be increased byhaving multiple execution candidates, from different reliabilitydomains, attempt to perform the RDMO in parallel. For example, assumethat the RDMO is to determine the average of a set of numbers. Inresponse to a single request from requesting entity 110, an executioncandidate implemented in NIC 102 and application 134 may both be invokedto perform the RDMO. In this example, both application 134 and theexecution candidate in NIC 102 would read the same set of values (e.g.data 140) from volatile memory 130, count the numbers in the set, sumthe numbers in the set, and then divide the sum by the count to obtainthe average. If neither execution candidate fails, then both executioncandidates may provide their response to requesting entity 110, whichmay simply discard the duplicate response.

In the case where one execution candidate fails and one succeeds, theexecution candidate that does not fail returns the response torequesting entity 110. Thus, the RDMO completes successfully despite thefact that something within machine 100 is not functioning correctly.

In the case where all execution candidates that initially attempted theRDMO fail, a second set of execution candidates, from differentreliability domains than the first set of execution candidates, may tryto execute the RDMO in parallel. In the case that at least one of thefallback execution candidate succeeds, the RDMO succeeds. This processmay proceed until one of the execution candidates for the RDMO succeeds,or all execution candidates for the RDMO fail.

Rather than have all successful execution candidates return responses torequesting entity 100, some “coordinating entity” within machine 100 mayinvoke a set of execution candidates to perform the RDMO in parallel. Ifmultiple candidates succeed, the successful candidates may provideresponses back to the coordinating entity which then returns a singleresponse to requesting entity 110. This technique simplifies the logicof the requesting entity 110, making transparent to requesting entity110 how many execution candidates on machine 100 were asked to performthe RDMO, and which of those execution candidates succeeded.

FIG. 3 is a flowchart illustrating use of parallel execution candidates,according to an embodiment. Referring to FIG. 3 , at step 300, arequesting entity sends a request for an RDMO to a remote computingdevice. At steps 302-1 through 302-N, each of N execution candidates forthe RDMO concurrently attempts to perform the RDMO. At step 304, if anyof the execution candidates succeeds, control passes to step 306 atwhich an indication of success (which may include additional resultsdata) is sent to the requesting entity. Otherwise, if all fail, anindication of failure is sent to the requesting entity at step 310.

Interpreter as a Nic-Based Execution Candidate

As mentioned above, one form of execution candidate for an RDMO is aninterpreter. For an interpreter to serve as an execution candidate forand RDMO, the interpreter is provided code which, when interpreted,executes the operations required by the RDMO. Such an interpreter may beexecuted, for example, by the processing unit 104 within NIC 102, byprocessor 120, or by a subset of the cores of processor 120.

According to one embodiment, the code for a particular RDMO isregistered with the interpreter. Once registered, requesting entitiesmay invoke the code (e.g. through an remote procedure call), causing thecode to be interpreted by the interpreter. In one embodiment, theinterpreter is a Java Virtual Machine, and the code is Java byte-code.However, the techniques used herein are not limited to any particulartype of interpreter or code. While an RDMO that is performed by aninterpreter within NIC 102 is likely to take much longer than the sameRDMO performed by a compiled application (e.g. application 134)executing on processor 120, the interpreter within NIC 102 may beoperational in periods during which some error prevents the operation ofthe compiled application.

Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 4 is a block diagram that illustrates a computersystem 400 upon which an embodiment of the invention may be implemented.Computer system 400 includes a bus 402 or other communication mechanismfor communicating information, and a hardware processor 404 coupled withbus 402 for processing information. Hardware processor 404 may be, forexample, a general purpose microprocessor.

Computer system 400 also includes a main memory 406, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 402for storing information and instructions to be executed by processor404. Main memory 406 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 404. Such instructions, when stored innon-transitory storage media accessible to processor 404, rendercomputer system 400 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 400 further includes a read only memory (ROM) 408 orother static storage device coupled to bus 402 for storing staticinformation and instructions for processor 404. A storage device 410,such as a magnetic disk, optical disk, or solid-state drive is providedand coupled to bus 402 for storing information and instructions.

Computer system 400 may be coupled via bus 402 to a display 412, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 414, including alphanumeric and other keys, is coupledto bus 402 for communicating information and command selections toprocessor 404. Another type of user input device is cursor control 416,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 404 and forcontrolling cursor movement on display 412. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 400 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 400 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 400 in response to processor 404 executing one or more sequencesof one or more instructions contained in main memory 406. Suchinstructions may be read into main memory 406 from another storagemedium, such as storage device 410. Execution of the sequences ofinstructions contained in main memory 406 causes processor 404 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical disks, magnetic disks, or solid-state drives, suchas storage device 410. Volatile media includes dynamic memory, such asmain memory 406. Common forms of storage media include, for example, afloppy disk, a flexible disk, hard disk, solid-state drive, magnetictape, or any other magnetic data storage medium, a CD-ROM, any otheroptical data storage medium, any physical medium with patterns of holes,a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip orcartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 402. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 404 for execution. For example,the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 400 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 402. Bus 402 carries the data tomain memory 406, from which processor 404 retrieves and executes theinstructions. The instructions received by main memory 406 mayoptionally be stored on storage device 410 either before or afterexecution by processor 404.

Computer system 400 also includes a communication interface 418 coupledto bus 402. Communication interface 418 provides a two-way datacommunication coupling to a network link 420 that is connected to alocal network 422. For example, communication interface 418 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 418 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 418sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 420 typically provides data communication through one ormore networks to other data devices. For example, network link 420 mayprovide a connection through local network 422 to a host computer 424 orto data equipment operated by an Internet Service Provider (ISP) 426.ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 428. Local network 422 and Internet 428 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 420and through communication interface 418, which carry the digital data toand from computer system 400, are example forms of transmission media.

Computer system 400 can send messages and receive data, includingprogram code, through the network(s), network link 420 and communicationinterface 418. In the Internet example, a server 430 might transmit arequested code for an application program through Internet 428, ISP 426,local network 422 and communication interface 418.

The received code may be executed by processor 404 as it is received,and/or stored in storage device 410, or other non-volatile storage forlater execution.

Cloud Computing

The term “cloud computing” is generally used herein to describe acomputing model which enables on-demand access to a shared pool ofcomputing resources, such as computer networks, servers, softwareapplications, and services, and which allows for rapid provisioning andrelease of resources with minimal management effort or service providerinteraction.

A cloud computing environment (sometimes referred to as a cloudenvironment, or a cloud) can be implemented in a variety of differentways to best suit different requirements. For example, in a public cloudenvironment, the underlying computing infrastructure is owned by anorganization that makes its cloud services available to otherorganizations or to the general public. In contrast, a private cloudenvironment is generally intended solely for use by, or within, a singleorganization. A community cloud is intended to be shared by severalorganizations within a community; while a hybrid cloud comprises two ormore types of cloud (e.g., private, community, or public) that are boundtogether by data and application portability.

Generally, a cloud computing model enables some of thoseresponsibilities which previously may have been provided by anorganization's own information technology department, to instead bedelivered as service layers within a cloud environment, for use byconsumers (either within or external to the organization, according tothe cloud's public/private nature). Depending on the particularimplementation, the precise definition of components or featuresprovided by or within each cloud service layer can vary, but commonexamples include: Software as a Service (SaaS), in which consumers usesoftware applications that are running upon a cloud infrastructure,while a SaaS provider manages or controls the underlying cloudinfrastructure and applications. Platform as a Service (PaaS), in whichconsumers can use software programming languages and development toolssupported by a PaaS provider to develop, deploy, and otherwise controltheir own applications, while the PaaS provider manages or controlsother aspects of the cloud environment (i.e., everything below therun-time execution environment). Infrastructure as a Service (IaaS), inwhich consumers can deploy and run arbitrary software applications,and/or provision processing, storage, networks, and other fundamentalcomputing resources, while an IaaS provider manages or controls theunderlying physical cloud infrastructure (i.e., everything below theoperating system layer). Database as a Service (DBaaS) in whichconsumers use a database server or Database Management System that isrunning upon a cloud infrastructure, while a DbaaS provider manages orcontrols the underlying cloud infrastructure, applications, and servers,including one or more database servers.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A method of attempting performance of anoperation on a particular computing device, comprising: concurrentlyexecuting, on the particular computing device that contains localvolatile memory: a first execution candidate, implemented in a firstreliability domain on the particular computing device, capable ofperforming the operation, a second execution candidate, implemented in asecond reliability domain on the particular computing device, capable ofperforming the operation, and a third execution candidate, implementedin a third reliability domain on the particular computing device,capable of performing the operation; wherein the first, second and thirdreliability domains are distinct from each other; wherein the operationrequires access to data in the local volatile memory associated with theparticular computing device; based on one or more factors, selecting atarget execution candidate, from among the first, second and thirdexecution candidates; and attempting performance of the operation usingthe target execution candidate.
 2. The method of claim 1, furthercomprising: receiving a request to perform the operation, from arequesting entity executing on a second computing device that is remoterelative to the particular computing device; wherein said attemptingperformance of the operation using the target execution candidate isperformed in response to receiving the request to perform the operationfrom the requesting entity.
 3. The method of claim 1, wherein saidselecting the target execution candidate is performed by a requestingentity executing on a second computing device that is remote relative tothe particular computing device.
 4. The method of claim 1, wherein saidselecting the target execution candidate is performed at the particularcomputing device.
 5. The method of claim 1, wherein the target executioncandidate is one of: an application running on one or more processors ofthe particular computing device; implemented in a network interfacecontroller of the particular computing device; implemented within anoperating system executing on the particular computing device;implemented within a privileged domain on the particular computingdevice; executed on a first set of one or more cores of a processor inthe particular computing device; executed on a second set of one or morecores of the processor, wherein membership of the second set of one ormore cores is different than membership of the first set of one or morecores; or an interpreter that interprets instructions that, wheninterpreted, cause performance of the operation.
 6. The method of claim5, wherein: the target execution candidate is an interpreter thatinterprets instructions that, when interpreted, cause performance of theoperation; the interpreter is implemented on a network interfacecontroller through which the particular computing device communicatesover a network; wherein the target execution candidate performs theoperation by interpreting instructions specified in data provided to thenetwork interface controller.
 7. A method of attempting performance ofan operation on a particular computing device, comprising: concurrentlyexecuting, on the particular computing device that contains localvolatile memory: a first execution candidate, implemented in a firstreliability domain on the particular computing device, capable ofperforming the operation, and a second execution candidate, implementedin a second reliability domain on the particular computing device,capable of performing the operation; wherein the first reliabilitydomain includes a first set of one or more cores of a processor of theparticular computing device; wherein the second reliability domainincludes a second set of one or more cores of the processor of theparticular computing device; wherein the first set of one or more coresis different than the second set of one or more cores; wherein theoperation requires access to data in the local volatile memory of theparticular computing device; selecting a target execution candidate,from among the first and second execution candidates; and causing thetarget execution candidate to perform the operation.
 8. The method ofclaim 7 wherein: the first execution candidate supports a first set ofoperations; the second execution candidate supports a second set ofoperations; and the second set of operations is a proper subset of thefirst set of operations.
 9. The method of claim 7, further comprising:receiving a request to perform the operation, from a requesting entityexecuting on a second computing device that is remote relative to theparticular computing device; wherein said attempting performance of theoperation using the target execution candidate is performed in responseto receiving the request to perform the operation from the requestingentity.
 10. One or more non-transitory computer-readable media storingone or more sequences of instructions for attempting performance of anoperation on a particular computing device, the one or more sequences ofinstructions comprising instructions that, when executed by one or moreprocessors, cause: concurrently executing, on the particular computingdevice that contains local volatile memory: a first execution candidate,implemented in a first reliability domain on the particular computingdevice, capable of performing the operation, a second executioncandidate, implemented in a second reliability domain on the particularcomputing device, capable of performing the operation, and a thirdexecution candidate, implemented in a third reliability domain on theparticular computing device, capable of performing the operation;wherein the first, second and third reliability domains are distinctfrom each other; wherein the operation requires access to data in thelocal volatile memory associated with the particular computing device;based on one or more factors, selecting a target execution candidate,from among the first, second and third execution candidates; andattempting performance of the operation using the target executioncandidate.
 11. The one or more non-transitory computer-readable media ofclaim 10, wherein the one or more sequences of instructions furthercomprise instructions that, when executed by one or more processors,cause: receiving a request to perform the operation, from a requestingentity executing on a second computing device that is remote relative tothe particular computing device; wherein said attempting performance ofthe operation using the target execution candidate is performed inresponse to receiving the request to perform the operation from therequesting entity.
 12. The one or more non-transitory computer-readablemedia of claim 10, wherein said selecting the target execution candidateis performed by a requesting entity executing on a second computingdevice that is remote relative to the particular computing device. 13.The one or more non-transitory computer-readable media of claim 10,wherein said selecting the target execution candidate is performed atthe particular computing device.
 14. The one or more non-transitorycomputer-readable media of claim 10, wherein the target executioncandidate is one of: an application running on one or more processors ofthe particular computing device; implemented in a network interfacecontroller of the particular computing device; implemented within anoperating system executing on the particular computing device;implemented within a privileged domain on the particular computingdevice; executed on a first set of one or more cores of a processor inthe particular computing device; executed on a second set of one or morecores of the processor, wherein membership of the second set of one ormore cores is different than membership of the first set of one or morecores; or an interpreter that interprets one or more instructions that,when interpreted, cause performance of the operation.
 15. The one ormore non-transitory computer-readable media of claim 14, wherein: thetarget execution candidate is an interpreter that interprets one or moreinstructions that, when interpreted, cause performance of the operation;the interpreter is implemented on a network interface controller throughwhich the particular computing device communicates over a network;wherein the target execution candidate performs the operation byinterpreting one or more instructions specified in data provided to thenetwork interface controller.
 16. One or more non-transitorycomputer-readable media storing one or more sequences of instructionsfor attempting performance of an operation on a particular computingdevice, the one or more sequences of instructions comprisinginstructions that, when executed by one or more processors, cause:concurrently executing, on the particular computing device that containslocal volatile memory: a first execution candidate, implemented in afirst reliability domain on the particular computing device, capable ofperforming the operation, and a second execution candidate, implementedin a second reliability domain on the particular computing device,capable of performing the operation; wherein the first reliabilitydomain includes a first set of one or more cores of a processor of theparticular computing device; wherein the second reliability domainincludes a second set of one or more cores of the processor of theparticular computing device; wherein the first set of one or more coresis different than the second set of one or more cores; wherein theoperation requires access to data in the local volatile memory of theparticular computing device; selecting a target execution candidate,from among the first and second execution candidates; and causing thetarget execution candidate to perform the operation.
 17. The one or morenon-transitory computer-readable media of claim 16 wherein: the firstexecution candidate supports a first set of operations; the secondexecution candidate supports a second set of operations; and the secondset of operations is a proper subset of the first set of operations. 18.The one or more non-transitory computer-readable media of claim 16,wherein the one or more sequences of instructions further compriseinstructions that, when executed by one or more processors, cause:receiving a request to perform the operation, from a requesting entityexecuting on a second computing device that is remote relative to theparticular computing device; wherein said attempting performance of theoperation using the target execution candidate is performed in responseto receiving the request to perform the operation from the requestingentity.